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FINAL ACTION 

Response to Amendment 

1 . The following Office Action is responsive to the amendment and remarks 
received on July 7, 2009. Claim 13 was cancelled and claims 15-17 are added. Claims 

1. 2, 4-7, 9, 11, 12, 14 and newly added claims 15-17 remain pending. 

Claim Rejections - 35 USC § 103 

2. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described 
as set forth in section 1 02 of this title, if the differences between the subject matter sought to be 
patented and the prior art are such that the subject matter as a whole would have been obvious 
at the time the invention was made to a person having ordinary skill in the art to which said 
subject matter pertains. Patentability shall not be negatived by the manner in which the 
invention was made. 

3. Claims 1-2, 5-7, 9, 11, 12, and 14-17 are rejected under 35 U.S.C. 103(a) as 
being unpatentable over the combination of Mitsui (US 7,222,235 B1), Hsieh et al. (US 
7,003,135 B2) and Rhoads (US 6,411,725 B1). 

Regarding claim 1, Mitsui discloses an information attaching/detecting system 
comprising: 

- object extracting means, for extracting a plurality of regions, from within the 
image (as depicted in figure 15, numeral 106 will perform the object extracting means, 

"[t]he image inputting apparatus 101(m) inputs an image and converts the input image 
to electronic data. The image inputting apparatus 101(m) may be a scanner or a digital 



Application/Control Number: 10/787,391 Page 3 

Art Unit: 2624 

camera. The image inputting apparatus 101(m) outputs tlie input image data as an 
original image A(m). Tine image processing apparatus 106(m) has a function of inputting 
the original image A(m) from the image inputting apparatus 101(m) and outputting a 
processed image C(m). The image processing apparatus 106(m) generates the 
processed image C(m) by embedding a digital watermark into the original image A(m)" 
at column 22, line 1 1 . The plurality of photographed objects is shown in figure 33, "FIG. 
33 shows how the digital watermark is embedded in the form Y ... the digital watermark 
is shaded for an explanation. For example, different kinds of digital watermarks are 
embedded for each of the region S1 and the region S2 because the character size is 
different between the region SI and the region S2 ... Furthermore, because the region 
S3 is a photograph, the digital watermark, which is different from in the region SI and 
the region S2, is embedded in the region S3" at column 38, line 56. See also figure 32 
which indicates a plurality of regions, such as character with REGION 1 and REGI0N2 
and photograph with REGION 3 and REGION n, is valid for object extraction); 

- information attaching means for attaching different information to each of the 
regions that do not overlap with other regions and acquiring said information-attached 
image (as discussed above and shown in figure 15, numeral 106 will perform the 
information attaching means. Different information attaches to each not overlap regions 
is shown in figure 33, "FIG. 33 shows how the digital watermark is embedded in the 
form Y ... the digital watermark is shaded for an explanation. For example, different 
kinds of digital watermarks are embedded for each of the region SI and the region S2 
because the character size is different between the region SI and the region S2 ... 
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Furthermore, because the region S3 is a photograph, the digital watermark, which is 
different from in the region SI and the region S2, is embedded in the region S3" at 
column 38, line 56. See also figure 32 which indicates a plurality of regions, such as 
character with REGION 1 and REGI0N2 and photograph with REGION 3 and REGION 
n, is valid for embedding information); 

- detection means for detecting said information from received data for each of 
said plurality regions contained in said information-attached image (as depicted in figure 
15, numeral 108 will detect the embedded information received from transmitted 
information-attached image. Figure 19 shows a block diagram for the information 
detection. Detailed discussion about the detection processing can be found from 
column 24, line 22 to column 25, line 8). 

Mitsui discloses that regions are used for information attaching/detecting. Mitsui 
does not explicitly disclose that the regions are human facial regions. Furthermore, 
Mitsui discloses the detection means from the transmitted information-attached image. 
Mitsui does not explicitly disclose that the received data can be a photographed-image 
by photographing an image reproducing medium on which information-attached image 
is reproduced. 

Hsieh, in the field of endeavor effaces tracking ("which is able to effectively 
detect and track a plurality of faces in real-time" at column 2, line 7), based on the fact 
that "[i]n order to efficiently separate the face regions from the complex backgrounds, a 
preferable approach is to use the hybrid information of color and motion. Then, the 
segmented regions are further verified to determine whether the correct faces are 
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extracted from the remaining distracters" at column 1 , line 42, teaches a methodology to 
track no-overlap human faces as shown in figure 1, numeral 10 and overlapped faces 
with: "a face-like region generator having a skin region extractor, a motion analyzer and 
a silhouette analyzer. The skin region extractor generates a plurality of skin regions by 
detecting skin color pixels of an input image. The motion analyzer determines possible 
face-like regions from the skin regions based on moving information of the input image. 
The silhouette analyzer analyzes whether there exists a raised shape in the image so 
as to separate connected regions ..." at column 2, line 39. Thus, Hsieh discloses a 
methodology to detect regions which are human facial regions for real-time tracking. 

It would have been obvious at the time the invention was made, that one of 
ordinary skill in the art would have been motivated to include the information 
attaching/detecting system Mitsui made with the application of face-like region 
generator as taught by Hsieh, in order to effectively tracking human faces in real-time as 
stated above by Hsieh. 

The combination of Mitsui and Hsieh does not explicitly disclose that the received 
data can be a photographed-image by photographing an image reproducing medium on 
which information-attached image is reproduced. 

Rhoads, in the field of endeavor of multimedia signal processing ("in particular 
relates to encoding information into and decoding information from video objects" at 
column 1 , line 30), teaches a robust data embedding methodology as shown in figure 2, 
"[i]n FIG. 2, physical objects 200 are pre-watermarked in a manner that survives the 
video capture process 202. For an example of a watermarking process that survives 
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digital to analog conversion (e.g., printing a digital image on a physical object), and then 
analog to digital conversion (e.g., capture via a video camera) ... The resulting video is 
then transmitted or broadcast 204" at column 7, line 29. Thus, the watermark can 
survive by printing this watermark embedded digital image and then capture this printed 
image via camera. Without departing from the scope and spirit of Rhoads' 
methodology, a second image captured by the camera can be an image from printing 
which created by Mitsui information attached first image. 

It would have been obvious at the time the invention was made, that one of 
ordinary skill in the art would have been motivated to enhance the information 
attaching/detecting system of the Mitsui and Hsieh combination, with the application of 
data embedding methodology as taught by Rhoads, so the embedded data can survive 
various scenarios as discussed above. 

Regarding claim 2, information attaching means is means for acquiring said 
information-attached image by hiddenly embedding said information in said image ("An 
image processing system can be provided such that the image providing apparatus 
recognizes a format of an image file, and provides the image file in which the digital 
watermark is embedded in a part of a region based on the format ..." at Mitsui column 5, 
line 61 . See also Rhoads figure 4, numeral 404 "...watermark encoding process 404 
encodes the auxiliary information into the content. ..." at Rhoads column 8, line 13). 
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Regarding claim 5, image pick-up means is a camera provided in a portable 
terminal (discussed in claim 1, digital camera can be the image pick-up means). 

Regarding claim 6, said information is location information representing storage 
locations of audio data correlated with said plurality of photographed objects ("Another 
aspect of the invention is a method for using a watermark that has been encoded into a 
video signal ... The watermark may include a direct (e.g., URL or network address) or 
indirect link (e.g., object identifier) to the web site ..." at Rhoads column 2, line 35), 
and which further comprises audio data acquisition means for acquiring said audio data, 
based on said location information (as depicted in Rhoads figure 1 A, numeral 114: "a 
user interface 1 14 executes and provides visual, audio, or audio-visual information to 
the user ... user interface receives input from the user, selecting a video object. In 
response, it performs an action associated with the selected object using the auxiliary 
object information decoded from the watermark ..." at Rhoads column 4, line 17. See 
also "The watermark may carry information or programmatic action. It may also link to 
external information or an action, such as retrieval and output of information stored 
elsewhere in a database, website, etc. Watermark linking enables the action associated 
with the watermark to be dynamic ..." at Rhoads column 4, line 29). 

Regarding claim 1 1 , the information device according to claim 1 , further 
comprising: 
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- a mobile communication terminal having a camera (as discussed in claim 1 and 
suggested by Rhoads a camera for the mobile communication terminal); 

- a server (as depicted in Rhoads figure 8, numeral 802 is a server); 

- wherein the camera of the mobile terminal photographs the image reproducing 
medium (as discussed in claim 1 and suggested by Rhoads a camera of the mobile 
communication terminal photographs the image reproducing medium); 

- wherein the input means and the detecting means are disposed in the server 
(as depicted in Rhoads figure 8, the server #802 receives input and parses and extracts 
to detect information: "The server, in response to receiving the message (828), parses it 
and extracts an index used to look up a corresponding action in a database (830) that 
associates many such indices to corresponding actions ..." at Rhoads column 14, line 
44). 

Regarding claim 12: - wherein the information is a location of audio data and 
the server acquires said audio data using said location and transmits said audio data to 
said mobile communication terminal (the combination of Mitsui and Rhoads teaches the 
mobile terminal about the location: "watermark may include a direct (e.g., URL or 
network address) or indirect link (e.g., object identifier) to the web site ..." at Rhoads 
column 2, line 44, to get audio data: "user interface 114 (Rhoads figure 1A) executes 
and provides visual, audio, or audio-visual information to the user ... user interface 
receives input from the user, selecting a video object. In response, it performs an action 
associated with the selected object ..." at column 4, line 17. Furthermore, "The server. 
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in response to receiving the message (828) (Rhoads figured), parses it and extracts an 
index used to \ook up a corresponding action in a database (830) that associates many 
such indices to corresponding actions ..." at Rhoads column 14, line 44). 

Regarding claim 14: - a region setting section, for setting block regions 
corresponding to each photographed object that do not overlap with other regions that 
include other photographed objects (as discussed in claim 1 , the figure 33 of Mitsui 
contains plurality of not overlapped regions with photographed objects such as SI , S2, 
S2, and S4); 

- an information attaching section, for attaching different information to each of 
the plurality of block regions which are set by the region setting means (as discussed in 
claim 1, the information attaching means for, information attached to each region is 
different). 

Regarding claim 15, wherein the plurality of facial regions are extracted by 
extracting skin-colored regions (as discussed in claim 1 by Hsieh for a skin region 
extractor). 

Regarding claim 16, wherein the plurality of facial regions are extracted by 
extracting face contours (as discussed in claim 1 by Hsieh for a silhouette analyzer). 
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Regarding claims 7 and 9: - extracting a plurality of facial regions of humans from 
within the image (discussed in claim 1, especially, Hsieh teaches the plurality of human 
facial regions extraction); 

- attaching different information to each of the plurality of facial regions in said 
image that respectively and acquiring said information-attached image (as discussed in 
claim 1 for information attaching means. See also Mitsui figure 33, "FIG. 33 shows how 
the digital watermark is embedded in the form Y ... the digital watermark is shaded for 
an explanation. For example, different kinds of digital watermarks are embedded for 
each of the region S1 and the region S2 because the character size is different between 
the region S1 and the region S2 ... Furthermore, because the region S3 is a 
photograph, the digital watermark, which is different from in the region SI and the 
region S2, is embedded in the region S3" at Mitsui column 38, line 56); 

- receiving photographed-image data obtained by photographing an image 
reproducing medium, on which the information-attached image is reproduced, with 
image pick-up means (as discussed in claim 1, input means for); 

- detecting said information from said photographed-image data for each of said 
plurality of facial regions contained in said information-attached image (as discussed in 
claimi, detection means for); 

- outputting the information corresponding to each of the plurality of facial 
regions, based on the detected information ("Wherever the video is distributed, a 
decoding process may be inserted to decode information about the video object from a 
watermark embedded in the video signal. This information may then be used to trigger 
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an action, sucli as fetcliing grapliics and displaying it to tlie user ..." at Rhoads column 
4, line 61). 

Regarding claim 17, wherein the outputting comprises providing a visual or audio 
display via at least one of a audio device and a display monitor (as discussed in claim 7 
for outputting detected information "may then be used to trigger an action, such as 
fetching graphics and displaying it to the user ..." at Rhoads column 4, line 64). 

4. Claim 4 is rejected under 35 U.S.C. 103(a) as being unpatentable over the 
combination of Mitsui, Hsieh, and Rhoads as applied to claim 1 discussed above, and 
further in view of Motta et al. (US 6,726,103 B1). 

The Mitsui, Hsieh, and Rhoads combination discloses an information 
attaching/detecting system with image pick-up means to receive photographed-image 
data. 

The Mitsui, Hsieh, and Rhoads combination does not explicitly disclose a way to 
correct the geometrical distorted photographed-image data. 

Motta, in the same field of endeavor of digital imaging ("relates to the field of 
image sensors and imaging systems" at column 1 , line 8), teaches a geometric 
correction processing system for the photographed-image data as depicted in figure 3. 
"Data processing may include but is not limited to data interpolation, noise reduction, 
color adjustment, and/or geometric corrections due to optical aberrations ..." at column 
4, line 4. 
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It would have been obvious at the time the invention was made, that one of 
ordinary skill in the art would have been motivated to provide the information 
attaching/detecting system made by Mitsui, Hsieh, and Rhoads combination with the 
geometrical distortion correction processor as taught by Motta, in order to provide "[a] 
system that can self test, detect errors, and correct such errors ... and would be of great 
value to many businesses and industries" at Motta column 2, line 29. 

Response to Arguments 

A) Summary of Applicant's Remark : 

The previous claim USC § 101 rejections should be withdrawn in view of the 
amendment. 
Examiner's Response : 

Examiner agrees, and the previous claim USC § 101 rejections are withdrawn. 

B) Summary of Applicant's Remark : 

"However, the regions S 1 , S2, and S4 shown in FIG. 33 are not regions which 
include a photographed object. That Is, this portion of Mitsui does not disclose 
extracting a plurality of regions, each of which includes a photographed object. Rather, 
FIG. 33 is an illustration showing how the digital watermark is embedded in form Y. 
However, form Y is not itself an image containing a plurality of photographed objects. 
Rather, form Y is a mere document format which only includes a single photograph. See 
col. 37, lines 40-46" at Remarks page 8, line 1 . 
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Examiner's Response : 

See Mitsui figure 32 which indicates a plurality of regions, such as character with 

REGION 1 and REGI0N2 and photograph with REGION 3 and REGION n, is valid for 

object extraction. Refer to the rejections above for further discussion. 

C) Summary of Applicant's Remark : 

"With regard to form Y, Mitsui merely discloses extracting a frame region of a 
photograph, not extracting a plurality of facial regions of humans. Nowhere does Mitsui 
disclose or fairly suggest extracting any facial regions. Therefore, Applicants submit 
Mitsui fails to disclose "object extracting means, for extracting a plurality of facial 
regions of humans from within the image; and information attaching means for attaching 
different information to each of the facial regions that do not overlap with other facial 
regions, and acquiring said information- attached image," as recited in claim 1" at 
Remarks page 8, line 7. 
Examiner's Response : 

Applicant's argument is moot in view of the new grounds of rejection advanced 
herein above. Specifically, the Hsieh et al. (US 7,003,135 B2) reference now teaches 
the concept of plurality of human facial regions extraction. Refer to the rejections above 
for further discussion. 

DJ Examiner's General Remarl<: 

Applicant has not submitted claims drawn to limitations, which define the 

operation and apparatus of Applicant's disclosed invention in manner, which 
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distinguishes over the prior art. As it is Applicant's right to claim as broadly as possible 
their invention, it is also the Examiner's right to interpret the claim language as broadly 
as possible. It is the Examiner's position that the detailed functionality that allows for 
Applicant's invention to overcome the prior art used in the rejection, fails to differentiate 
in detail how these features of applicant's specification are. It is suggested that 
Applicant compare the original specification and claim language with the cited prior art 
used in the rejection section above to draw an amended claim set to further the 
prosecution. Examiner reiterates the need for the Applicant to more clearly and 
distinctly define the claimed invention. 

Conclusion 

5. Applicant's amendment is rejected in this Office action. Accordingly, THIS 
ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the 
extension of time policy as set forth in 37 CFR 1 .136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 
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6. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Eueng-nan Yeh whose telephone number is 571-270- 
1586. The examiner can normally be reached on Monday-Friday 8AM-4:30PM EDT. 

7. If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Vikkram Bali can be reached on 571-272-7415. The fax phone number for 
the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

Eueng-nan Yeh 
Assistant Patent Examiner 
Art Unit: 2624 
/E.Y./ 

A/IKKRAM BALI/ 

Supervisory Patent Examiner, Art Unit 2624 



